Pose Robust Video - Based Face Recognition

Researchers have been working on human face recognition for decades. Face recognition is hard due to different types of variations in face images, such as pose, illumination and expression, among which pose variation is the hardest one to deal with. To improve face recognition, this thesis presents an integrated approach to performing pose robust video-based face tracking and recognition by using a face mosaic model. We approximate a human head with a 3D ellipsoid model, where each face image is a projection of the 3D ellipsoid at a certain pose. In our approach, both training and test images are projected back to the surface of the 3D ellipsoid, according to their estimated poses, to form the texture maps. Thus the recognition can be conducted by comparing texture maps instead of the original images, as done in traditional face recognition. In addition, by representing the texture map as an array of local patches, we can train a probabilistic model for comparing corresponding patches. With multiple training images under different views, we are able to obtain a statistical mosaic model as well as a geometric deviation model, which not only reduces the blurring effect in the mosaic model, but also serves as an indication of how much the actual human faces geometry deviates from the 3D ellipsoid model. Furthermore, we apply the face mosaic model to video-based face recognition. The mosaic model is able to simultaneously track, register, and recognize human faces from video sequences. Finally, we also apply the updating-during-recognition scheme in using the mosaic model. This scheme allows the mosaic model to be updated during the test stage in order to enhance the modeling and recognition over time.